{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# HW 1: System Prompt Engineering with Phoenix \n",
    "\n",
    "## 🎯 Assignment Overview\n",
    "\n",
    "Welcome to Homework Assignment 1! In this assignment, we'll be building the foundation for our recipe chatbot by focusing on prompt engineering and systematic testing. Our goal is to get a working prompt for us to use in Homework 2. \n",
    "\n",
    "### What We'll Accomplish:\n",
    "\n",
    "1. **🤖 Write an Effective System Prompt**: Create a well-crafted system prompt that defines our recipe bot's personality, capabilities, and output format\n",
    "2. **📊 Expand Our Query Dataset**: Add diverse test queries to evaluate different aspects of our chatbot's performance  \n",
    "3. **🔬 Set up Phoenix Testing**: Use Phoenix to test and improve our prompt.\n",
    "\n",
    "Let's dive in! 🚀\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install required packages\n",
    "import subprocess\n",
    "import sys\n",
    "\n",
    "\n",
    "def install_packages():\n",
    "    packages = [\n",
    "        \"arize-phoenix\",\n",
    "        \"openai\",\n",
    "        \"pandas\",\n",
    "        \"openinference-instrumentation-openai\",\n",
    "        \"nest-asyncio\",\n",
    "    ]\n",
    "\n",
    "    for package in packages:\n",
    "        print(f\"Installing {package}...\")\n",
    "        subprocess.check_call([sys.executable, \"-m\", \"pip\", \"install\", package])\n",
    "\n",
    "\n",
    "# Uncomment to install packages\n",
    "# install_packages()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/sallyanndelucia/miniconda3/envs/dev_testing/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ All imports successful!\n"
     ]
    }
   ],
   "source": [
    "import nest_asyncio\n",
    "import pandas as pd\n",
    "\n",
    "from phoenix.client import AsyncClient\n",
    "from phoenix.client.types import PromptVersion\n",
    "\n",
    "# Enable nested async for Jupyter\n",
    "nest_asyncio.apply()\n",
    "\n",
    "print(\"✅ All imports successful!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "## 1. Environment Setup \n",
    "\n",
    "Before we start prompt engineering, we need to set up Phoenix on our local machine. You can either do this in the notebook with the below code or from the terminal using `phoenix serve` or `python -m phoenix.server.main serve`\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Launch Phoenix locally in the notebook\n",
    "# session = px.launch_app(use_temp_dir=False)\n",
    "# print(f\"🔥 Phoenix is running at: {session.url}\")\n",
    "# print(\"📱 Click the link above to open the Phoenix UI in your browser\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "## 2. Prompt Management - Create and Store Prompts\n",
    "\n",
    "Now let's create a prompt for our recipe chat bot and store them in Phoenix's prompt management. We'll start with a super basic prompt that we can iterate on later. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "🎯 Prompt created! You can now view it in the Phoenix UI under the 'Prompts' section.\n"
     ]
    }
   ],
   "source": [
    "# Create recipe assistant prompts\n",
    "recipe_prompt_v1 = \"\"\"\n",
    "You are a recipe assistant. Your job is to generate easy to follow recipes and cooking advice. You should always provide ingredient lists with precise measurements using standard units. You should always include clear, step-by-step instructions.\n",
    "\n",
    "User Query: {query}\n",
    "\n",
    "\"\"\"\n",
    "\n",
    "prompt_name = \"recipe-assistant-v1\"\n",
    "px_client = AsyncClient()\n",
    "prompt = await px_client.prompts.create(\n",
    "    name=prompt_name,\n",
    "    prompt_description=\"Basic recipe assistant prompt\",\n",
    "    version=PromptVersion(\n",
    "        [{\"role\": \"system\", \"content\": recipe_prompt_v1}],\n",
    "        model_name=\"gpt-4o-mini\",\n",
    "    ),\n",
    ")\n",
    "\n",
    "\n",
    "print(\"\\n🎯 Prompt created! You can now view it in the Phoenix UI under the 'Prompts' section.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "## 3. Create Dataset from CSV\n",
    "\n",
    "Let's create a dataset with some recipe queries and then add 10 new ones.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>query</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Suggest a quick vegan breakfast recipe</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>I have chicken and rice. what can I cook?</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Give me a dessert recipe with chocolate</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id                                      query\n",
       "0   1     Suggest a quick vegan breakfast recipe\n",
       "1   2  I have chicken and rice. what can I cook?\n",
       "2   3   Give me a dessert recipe with chocolate "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"data/sample_queries.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Expanding Our Test Dataset\n",
    "\n",
    "Now let's add more diverse queries to test different aspects of our recipe chatbot. According to the homework assignment, we need at least 10 new queries that cover:\n",
    "\n",
    "- **Specific cuisines** (Italian, Thai, etc.)\n",
    "- **Dietary restrictions** (vegan, gluten-free)  \n",
    "- **Available ingredients** (\"What can I make with X?\")\n",
    "- **Meal types** (breakfast, lunch, dinner, snacks)\n",
    "- **Time constraints** (\"under 30 minutes\")\n",
    "- **Skill levels** (beginner-friendly)\n",
    "- **Vague queries** (to test how the bot handles ambiguity)\n",
    "\n",
    "> **Note**: You can also add queries directly in the Phoenix UI, but we'll do it programmatically here for reproducibility."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "📤 Uploading dataset...\n",
      "💾 Examples uploaded: http://127.0.0.1:6006/datasets/RGF0YXNldDoy/examples\n",
      "🗄️ Dataset version ID: RGF0YXNldFZlcnNpb246Mg==\n"
     ]
    }
   ],
   "source": [
    "# Add 10 new queries to the dataset\n",
    "new_queries = [\n",
    "    \"How do I make sushi rice?\",\n",
    "    \"What's the secret to fluffy pancakes?\",\n",
    "    \"How do I properly season a cast iron pan?\",\n",
    "    \"What's the best way to cook salmon?\",\n",
    "    \"How do I make fresh pasta sauce?\",\n",
    "    \"What spices are essential for Indian cooking?\",\n",
    "    \"How do I make sourdough starter?\",\n",
    "    \"What's the difference between baking and broiling?\",\n",
    "    \"How do I make homemade stock?\",\n",
    "    \"What's the best way to store fresh herbs?\",\n",
    "]\n",
    "\n",
    "new_df = pd.DataFrame(\n",
    "    {\n",
    "        \"query\": new_queries,\n",
    "    }\n",
    ")\n",
    "\n",
    "# Append to existing dataset\n",
    "full_df = pd.concat([df, new_df], ignore_index=True)\n",
    "\n",
    "px_client = AsyncClient()\n",
    "dataset = await px_client.datasets.create_dataset(\n",
    "    dataframe=full_df,\n",
    "    name=\"recipe-queries\",\n",
    "    input_keys=[\"query\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing Our Prompt in Phoenix\n",
    "\n",
    "Now that we have our prompt and dataset uploaded to Phoenix, it's time to test and iterate!\n",
    "\n",
    "### 🧪 Testing in the Phoenix UI:\n",
    "\n",
    "1. **Navigate to Phoenix**\n",
    "2. **Find Your Prompt**: Go to the \"Prompts\" section and find your `recipe-assistant-v1` prompt  \n",
    "3. **Create an Experiment**: Use your `recipe-queries` dataset to test the prompt\n",
    "4. **Evaluate Results**: Look for:\n",
    "   - Proper markdown formatting\n",
    "   - Clear ingredient lists and instructions\n",
    "   - Appropriate creativity vs. accuracy balance\n",
    "   - Handling of edge cases (vague queries, dietary restrictions, etc.)\n",
    "\n",
    "### 🔍 What to Look For:\n",
    "- **Formatting**: Are recipes properly structured with `##` headers and `###` sections?\n",
    "- **Completeness**: Do responses include ingredients, instructions, and helpful tips?\n",
    "- **Safety**: Are unsafe requests properly declined?\n",
    "- **Creativity**: Does the bot provide helpful variations and substitutions?\n",
    "- **Consistency**: Are measurements in standard units and instructions clear?\n",
    "\n",
    "### ✨ Iteration Tips:\n",
    "- Test edge cases: very vague queries, dietary restrictions, cooking methods\n",
    "- Try different phrasings of the same request\n",
    "- Look for patterns in failures and adjust your prompt accordingly\n",
    "- Use Phoenix's evaluation tools to systematically measure performance\n",
    "\n",
    "After testing several queries and feeling confident about your prompt's performance, move on to applying it to the local codebase!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "raw"
    }
   },
   "source": [
    "## 4. Assignment Complete! \n",
    "\n",
    "Excellent work! We've successfully completed Homework 1 by building and testing our recipe chatbot prompt entirely within the Phoenix platform.\n",
    "\n",
    "### 🎯 What We Accomplished:\n",
    "- ✅ Created a comprehensive system prompt with clear formatting guidelines\n",
    "- ✅ Set up Phoenix for testing and prompt management  \n",
    "- ✅ Built and uploaded a diverse test dataset with 13+ queries\n",
    "- ✅ Tested our prompt systematically in the Phoenix UI\n",
    "- ✅ Iterated on prompt performance using Phoenix's evaluation tools\n",
    "- ✅ Established LLMOps best practices for prompt development\n",
    "\n",
    "\n",
    "\n",
    "### 🚀 Ready for Homework 2!\n",
    "With our solid foundation of prompt engineering and systematic evaluation, we're now prepared to tackle Homework 2. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dev_testing",
   "language": "python",
   "name": "dev_testing"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
